1/5 




Agent 1 

4 



Agent 2 



110 
Network 



Agent 1 



Agent 2 



12 



Se 



Fi G. 1 PRIOR ART 



200 


User- Agent : 


Robot_ 1 




Disallow : 


/england /london 




Disallow 


/ trance / paris 



FIG. 2 



PRIOR ART 




FIG. 3 PRIOR ART 



(38$ 20010032 US1 



7 IJ 




Session management system generates 'session ID' 




r 


Web page configured to support hidden URLS 










'Session ID 'and a v page ID 'encoded 


410 




dynamically into the hidden link 



400 
405 



417 



1 



^Session ID 'encoded 
dynamically into 
further hidden links 




A robot extracts ALL links by making requests 



420 



The robot passes links to the 
controlling application 



425 



The application decides which links to follow 



I 



430 



The server logs all requests 
(and information) made to it 



435 



The server stores all th e data in a log file 




440 



END 



FIG. U 



(33920010032 US1 



4/5 



500 



Log file analysed 



led? 



:"r. 



Blind pages associated with 505 

a session ID identified 



Session ID given a score 

I 



Score is used by a robot 
identification rule 



510 



515 




Associate the session ID with a 
robot with a probabflity value 



FIG. 5 



GB9 20010032 US1 



5/5 



,4 



Identified robot 
is given a profile 



I 



600 



Fields in profile used 
in robot management rules 




Invoke default 
'Rule 4' 




605 



640 



615 



Invoke 
'Rule T 



625 



Invoke 
'Rule 2' 



635 




END 



FIG. 6 



